[201803] Set a rate limit on syslog messages from all Docker containers #2574
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can you please explain why we need to do this? |
|
@nikos-github: This is to prevent overloading remote syslog servers if a process starts spamming the syslog. For example, if certain ASICs present a parity error that does not get auto-corrected, the syncd container can be responsible for sending > 200,000 syslog messages every 5 minutes. |
|
@jleveque What is the effect of this on debug level setting on daemons in containers? How did you determine the numbers? Are they meant to be per PID? If so, it implies that we think 30 msgs per second for a PID is the limit. Is that actually enough at boot time? Why not implement the rate limiting at the source/syncd for the example event that you mentioned or filter it and not send it to syslog server once you know what to look for? I'm not convinced this is the right approach since there are valid cases where we may want to generate and see more than 30 msgs per second from a PID. |
Debug level settings were not taken into effect for this PR. This PR is based off the level of log messages of an image built with current default settings.
The numbers are not per-PID, but rather per-container. The numbers were an average of a few switches configured with 120+ ports. During normal operation, the highest rate of messages occurs at boot. I found with these devices, the average maximum rate of total messages (base image + all containers) is ~12,000 per 5 minute interval. The average maximum rate per container at boot is as follows:
Thus, setting the per-container rate limit to 10,000 messages per 5-minute interval should not interfere with boot messages. If we are concerned at all, I think we could increase the limit to 20,000 or even a bit higher. The idea here is simply to prevent a rogue process from spamming a remote syslog server, so there is a bit of flexibility. |
lguohan
approved these changes
Feb 25, 2019
vivekrnv
added a commit
to vivekrnv/sonic-buildimage
that referenced
this pull request
Dec 17, 2022
6185324 dereg acl-rule counters during acl-table del (sonic-net#2574) b865352 Align watermark flow with port configuration (sonic-net#2525) f2d2fb3 L3 / L3 V6 Egress ACL table creation failure (sonic-net#2561) 577f696 [muxorch] Skip programming ACL for standby `active-active` ports (sonic-net#2569) 242ee11 [muxorch] Skip programming SoC IP kernel tunnel route (sonic-net#2557) 6695113 [gearbox] Support setting tx taps on gearbox ports (sonic-net#2158) 872f7bf [portinit] Do not call GET on SAI_PORT_ATTR_SPEED when AUTONEG is enabled (sonic-net#2484) 6afefe1 [vstest][virtual chassis] Removed dvs.runcmd using click commands (sonic-net#2214) b8521cc [p4orch]: PINS Extension tables support (sonic-net#2506) d0419dc sonic-swss: Fix orchagent crash in generateQueueMapPerPort. (sonic-net#2552) bd652a0 [muxorch] Adding case for maintaining current state (sonic-net#2280) 6b6dda6 [Centec]for support mclag of centec to configure port isolate-group sonic-net#2529 ec507a4 [ACL] Support ACTION_COUNTER action in custom ACL table type (sonic-net#2550) 1a74604 Use github code scanning instead of LGTM (sonic-net#2546) bc3c894 [dual-tor] add missing SAI attribte in order to create IPNIP tunnel (sonic-net#2503) dca78d8 (origin/202211) [Fdbsyncd] Bug Fix for remote MAC move to local MAC and Fix for Static MAC advertisement in EVPN. (sonic-net#2521) Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>
liat-grozovik
pushed a commit
that referenced
this pull request
Dec 20, 2022
6185324 dereg acl-rule counters during acl-table del (#2574) b865352 Align watermark flow with port configuration (#2525) f2d2fb3 L3 / L3 V6 Egress ACL table creation failure (#2561) 577f696 [muxorch] Skip programming ACL for standby `active-active` ports (#2569) 242ee11 [muxorch] Skip programming SoC IP kernel tunnel route (#2557) 6695113 [gearbox] Support setting tx taps on gearbox ports (#2158) 872f7bf [portinit] Do not call GET on SAI_PORT_ATTR_SPEED when AUTONEG is enabled (#2484) 6afefe1 [vstest][virtual chassis] Removed dvs.runcmd using click commands (#2214) b8521cc [p4orch]: PINS Extension tables support (#2506) d0419dc sonic-swss: Fix orchagent crash in generateQueueMapPerPort. (#2552) bd652a0 [muxorch] Adding case for maintaining current state (#2280) 6b6dda6 [Centec]for support mclag of centec to configure port isolate-group #2529 ec507a4 [ACL] Support ACTION_COUNTER action in custom ACL table type (#2550) 1a74604 Use github code scanning instead of LGTM (#2546) bc3c894 [dual-tor] add missing SAI attribte in order to create IPNIP tunnel (#2503) dca78d8 (origin/202211) [Fdbsyncd] Bug Fix for remote MAC move to local MAC and Fix for Static MAC advertisement in EVPN. (#2521)
dprital
added a commit
to dprital/sonic-buildimage
that referenced
this pull request
Jan 3, 2023
Update sonic-swss submodule pointer to include the following: * fb68ca8 [202205] [ACL] dereg acl-rule counters during acl-table del (sonic-net#2574) ([sonic-net#2582](sonic-net/sonic-swss#2582)) * 8c9092c [202205] Fixed set mtu for deleted subintf due to late notification ([sonic-net#2595](sonic-net/sonic-swss#2595)) Signed-off-by: dprital <drorp@nvidia.com>
8 tasks
liat-grozovik
pushed a commit
that referenced
this pull request
Jan 4, 2023
* [202205][submodule] Advance sonic-swss pointer Update sonic-swss submodule pointer to include the following: * fb68ca8 [202205] [ACL] dereg acl-rule counters during acl-table del (#2574) ([#2582](sonic-net/sonic-swss#2582)) * 8c9092c [202205] Fixed set mtu for deleted subintf due to late notification ([#2595](sonic-net/sonic-swss#2595)) Signed-off-by: dprital <drorp@nvidia.com>
dprital
added a commit
to dprital/sonic-buildimage
that referenced
this pull request
Jan 4, 2023
Update sonic-utilities submodule pointer to include the following: * 3cb66b4 [202205] Preserve copp tables through DB migration (sonic-net#2524) ([sonic-net#2574](sonic-net/sonic-utilities#2574)) Signed-off-by: dprital <drorp@nvidia.com>
liat-grozovik
pushed a commit
that referenced
this pull request
Jan 4, 2023
…13247) Update sonic-utilities submodule pointer to include the following: 3cb66b4 [202205] Preserve copp tables through DB migration (2524) (#2574) Update sonic-swss submodule pointer to include the following: c9ca7c8 Modify coppmgr mergeConfig to support preserving copp tables through reboot. (#2548) Signed-off-by: dprital <drorp@nvidia.com>
8 tasks
StormLiangMS
added a commit
that referenced
this pull request
Feb 17, 2023
Why I did it Submodule advances: sonic-utilities 8e8e6088 - [202211][dhcp_relay] Remove add field of vlanid to DHCP_RELAY table while adding vlan ([201811 sub-module] advance sub-modules: utilities, swss, swss-common #2679) (16 hours ago) [Yaqiang Zhu] 1400fb94 - [GCU] Ignore bgpraw in GCU applier (Fix sfputil indexing for 7170-Q59S20 #2623) (15 hours ago) [jingwenxie] f76a6364 - [vlan] Refresh dhcpv6_relay config while adding/deleting a vlan ([sonic-py-swsssdk] Update submodule #2660) (15 hours ago) [Yaqiang Zhu] 7849e18d - [db_migrator] make LOG_LEVEL_DB migration more robust (Mellanox platform: attach queues 2 and 6 to lossy profile using generic buffer template #2651) (16 hours ago) [Stepan Blyshchak] c7df6dfa - Fixed a bug in "show vnet routes all" causing screen overrun. (Add hook to allow customizing link cable lengths #2644) (16 hours ago) [siqbal1986] a5505f02 - show logging CLI support for logs stored in tmpfs (Traceback error seen while issuing show interface commands with if_names #2641) (16 hours ago) [mihirpat1] bbacb91a - [system-health] Fix issue: show system-health CLI crashes (Updating deb package for platform and sai #2635) (16 hours ago) [Junchao-Mellanox] 8d724024 - [sai_failure_dump]Invoking dump during SAI failure ([dockers]: Upgrade LLDP docker to stretch build #2633) (16 hours ago) [Sudharsan Dhamal Gopalarathnam] 3c3be526 - Add transceiver info CLI support to show output from TRANSCEIVER_INFO for ZR ([submodule]: Update sonic-sairedis pointer #2630) (16 hours ago) [mihirpat1] 37f41666 - [show] add support for gRPC show commands for active-active ([bitmap-vnet]: Bitmap vnet test image [DO NOT MERGE] #2629) (16 hours ago) [vdahiya12] b06d7fe4 - [show_bfd] add local discriminator in show bfd command ([Pmon] Selectively load pmon container daemons #2625) (16 hours ago) [Baorong Liu] 6adcd3e8 - [GCU] Ignore bgpraw table in GCU operation ([Mellanox] Fix SAI version #2628) (16 hours ago) [jingwenxie] c65bdc35 - [muxcable][config] Add support to enable/disable ceasing to be an advertisement interface when radv service is stopped (Add knob in ConfigDB to enable/disable telemetry container #2622) (16 hours ago) [Jing Zhang] 91e9457f - Add Transceiver PM basic CLI support to show output from TRANSCEIVER_PM table for ZR ([201803] Restart SwSS, syncd and dependent services if a critical process in syncd container exits #2615) (16 hours ago) [longhuan-cisco] 54cc8c5a - Remove TODO comment which is no longer relevant (Warm-reboot: teamd warm restart caused neighbor deleted and learned again. #2600) (16 hours ago) [Lior Avramov] 6891b4fb - Making 'show feature autorestart' more resilient to missing auto_restart config in CONFIG_DB ([submodule] update mellanox hw-mgmgt pointer (V.2.0.0061) #2592) (16 hours ago) [kartik-arista] 1e8bea37 - [storyteller] add link prober state change to story teller ([sonic-buildimage] New feature managementVRF(L3mdev) #2585) (16 hours ago) [Jing Zhang] 7481a20f - Extend fast-reboot STATE_DB entry timer ([submodule]: update sonic-swss-common, sonic-py-swsssdk, sonic-snmpagent #2577) (16 hours ago) [Aryeh Feigin] 0e08701c - [sonic_installer] use /etc/resolv.conf from the host when migrating packages (Set a rate limit on syslog messages from all Docker containers #2573) (16 hours ago) [Stepan Blyshchak] 06096780 - Fixed admin state config CLI for Backport interfaces (Prior to install a new ONIE SONiC image, delete all partitions except EFI/ONIE #2557) (16 hours ago) [anamehra] 9f1f13e4 - [show] Add bgpraw to show run all (Fixed typo on paragraph #40 #2537) (16 hours ago) [jingwenxie] 98bc8bd2 - [chassis][voq] Add "show fabric reachability" command. ([ntp]: Build 4.2.6 locally. #2528) (16 hours ago) [jfeng-arista] 3a50b63f - Preserve copp tables through DB migration ([docker-radvd]: upgrade docker radvd to stretch based #2524) (16 hours ago) [Aryeh Feigin] 28f6b127 - [masic] 'show interfaces counters' reminds to use '-d all' option to check for internal links (solve dependency issue #2466) (16 hours ago) [wenyiz2021] 15026e14 - suppport multi asic for show queue counter ([dockers] Prevent old supervisord messages from gettting re-logged to syslog #2439) (16 hours ago) [zhixzhu] 2d773e17 - [masic support] 'show run bgp' support for multi-asic (lo address not synced to the asic #2427) (16 hours ago) [wenyiz2021] sonic-swss 4f304bc - [EVPN]Handling race condition when remote VNI arrives before tunnel map entry ([sonic-quagga] Function defect, do NOT cancel route while connect IP down #2642) (15 hours ago) [Sudharsan Dhamal Gopalarathnam] 34fc615 - [sai_failure_dump]Invoking dump during SAI failure (Add hook to allow customizing link cable lengths #2644) (15 hours ago) [Sudharsan Dhamal Gopalarathnam] b817695 - [autoneg]Fixing adv interface types to be set when AN is disabled (Fix issue with platform file path name #2638) (15 hours ago) [Sudharsan Dhamal Gopalarathnam] ab36bd4 - [bfdorch] add local discriminator to state DB ([bitmap-vnet]: Bitmap vnet test image [DO NOT MERGE] #2629) (15 hours ago) [Baorong Liu] 6343471 - Remove TODO comments that are no longer relevant (Add knob in ConfigDB to enable/disable telemetry container #2622) (15 hours ago) [Lior Avramov] 2b1869c - [refactor]Refactoring sai handle status (Rollback kernel submodule update. #2621) (15 hours ago) [Sudharsan Dhamal Gopalarathnam] c41a1b7 - Fix issue ARP entry is out of sync between kernel and APPL_DB after warm reboot if the ARP entry is updated more than once during warm reboot in PFC watchdog warm reboot test #13341 ARP entry can be out of sync between kernel and APPL_DB if multiple updates are received from RTNL ([sub module] advance sonic-utilities sub module for 201811 branch #2619) (15 hours ago) [Stephen Sun] da0cf7a - Changed the BFD default detect multiplier to 10x ("failed to load plugin io.containerd.snapshotter..." seen during linux boot up #2614) (15 hours ago) [siqbal1986] 13b5adf - [vstest] Only collect stdout of orchagent_restart_check in vstest ([submodules] update swss and utilities pointers #2597) (15 hours ago) [bingwang-ms] 2b9d94d - Avoid aborting orchagent when setting TUNNEL attributes (build failing for PLATFORM=p4 #2591) (15 hours ago) [Stephen Sun] 99b7d3b - Only collect stdout of orchagent_restart_check in vstest ( [saibcm-modules]: import new bcm modules #2578) (15 hours ago) [bingwang-ms] 5209c42 - dereg acl-rule counters during acl-table del ([201803] Set a rate limit on syslog messages from all Docker containers #2574) (15 hours ago) [Vivek] ae68054 - Fixed set mtu for deleted subintf due to late notification ([vs]: Add option to specify platform name for DVS orchagent #2571) (15 hours ago) [EdenGri] ab13dfa - Remove TODO comments which are no longer needed (support set timezone in ConfigDB #2568) (15 hours ago) [Junchao-Mellanox] a3545cf - Modify coppmgr mergeConfig to support preserving copp tables through reboot. (Added new SN3700/SN3700C Mellanox platforms #2548) (15 hours ago) [Aryeh Feigin] be16e79 - Use github code scanning instead of LGTM ([201803] [services] Restart SwSS service upon unexpected critical process exit #2546) (15 hours ago) [Liu Shilong] 63c0234 - Updated handling of VRF_VNI mapping and VLAN_VNI mapping for same VNI ID (Move warm_restart enable/disable config to stateDB WARM_RESTART_ENABL… #2538) (15 hours ago) [Tapash Das] 4844111 - Fix potential risks ([mlnx] Fix sai xml path for boxer platform #2516) (15 hours ago) [Liran-Ar] 6420808 - [p4orch]: PINS Extension tables support ([build] When generating image version, handle case where current commit has no reachable tags #2506) (15 hours ago) [svshah-intel] sonic-swss-common 1badd46 - Increase the netlink buffer size from 3MB to 16MB. (arp_update doesn't sleep 300 between each execution #739) (14 hours ago) [KISHORE KUNAL] 6555057 - Refactor eventpublisher deinit ([acl] Add default deny rule for l3 table #734) (14 hours ago) [Zain Budhwani] f4d6de7 - Use github code scanning instead of LGTM ([sonic-quagga]:update submodule #718) (14 hours ago) [Liu Shilong] sonic-linux-kernel 74f9a8f - Update linux kernel for hw-mgmt V.7.0020.4104 (Move template files to /usr/share/sonic/templates #305) (14 hours ago) [Stephen Sun] 6365701 - Fixes for emmc unreliability ([build_debian.sh]: Integrate system dump script #270) (14 hours ago) [Samuel Angebault] How I did it How to verify it
mihirpat1
pushed a commit
to mihirpat1/sonic-buildimage
that referenced
this pull request
Jun 14, 2023
*Deleting ACL Tables deletes all the associated Rules but it doesn't delete the associated counters.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Enable rate limiting on rsyslog messages originating from each Docker container. Each container is limited to 10,000 messages per 5-minute interval.